Skip to content

Add wait time after scaling up/down operations to ensure logs are uploaded to CW #5876

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 1 commit into
base: develop
Choose a base branch
from
Draft

Add wait time after scaling up/down operations to ensure logs are uploaded to CW #5876

wants to merge 1 commit into from

Conversation

EddyMM
Copy link
Contributor

@EddyMM EddyMM commented Nov 22, 2023

Description of changes

  • When scaling up or down, we run assertions on the scaling metrics after a fixed amount of time
  • If any of the assertions fail, the test fails and the cluster is deleted as part of the test teardown.
  • The CW logs are preserved but they may not have captured some relevant info about the compute nodes that were launched e.g. nodes with bootstrap errors.
  • The changes in this PR add a wait time before we run assertions on the collected scaling metrics to ensure we can review logs linked to the assertions in case of a failure

Tests

  • Ran test with a manually injected assertion failure to ensure we receive logs that are at least up to the timestamp of the latest scaling metric used in the assertion

References

  • N/A

Checklist

  • Make sure you are pointing to the right branch.
  • If you're creating a patch for a branch other than develop add the branch name as prefix in the PR title (e.g. [release-3.6]).
  • Check all commits' messages are clear, describing what and why vs how.
  • Make sure to have added unit tests or integration tests to cover the new/modified code.
  • Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@EddyMM EddyMM added the skip-changelog-update Disables the check that enforces changelog updates in PRs label Nov 22, 2023
@EddyMM EddyMM force-pushed the wip/add-wait-time-for-log-upload branch from c757bf8 to db1f426 Compare November 22, 2023 14:13
@codecov Codecov
Copy link

codecov bot commented Nov 22, 2023 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (c3f620c) 90.13% compared to head (b33f0a4) 90.13%.
Report is 2 commits behind head on develop.

Additional details and impacted files 
@@           Coverage Diff            @@
##           develop    #5876   +/-   ##
========================================
  Coverage    90.13%   90.13%           
========================================
  Files          180      180           
  Lines        15725    15725           
========================================
  Hits         14173    14173           
  Misses        1552     1552
Flag Coverage Δ
unittests 90.13% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@EddyMM EddyMM force-pushed the wip/add-wait-time-for-log-upload branch from db1f426 to b33f0a4 Compare November 22, 2023 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
skip-changelog-update Disables the check that enforces changelog updates in PRs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@EddyMM